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ABSTRACT 

This study examined and refined a mathematics skills 
instrument used to predict success in an introductory statistics 
course and to measure improvement in mathematics skills. Alternate 
forms of a 28-item true-false test were administered as a pretest and 
posttest to 18 undergraduate and 19 graduate students enrolled in 
introductory educational statistics courses at the University of 
Alabama. Cronbach's alpha and corrected item-total correlations were 
computed for Lhe pretest. Item analysis resulted in the elimination 
of 10 items from the original instrument. The alpha coefficient for 
the resulting 18-item instrument was .75. Total pretest scores were 
correlated with each of three course exams and with course average. 
Results revealed significant correlations with examinations 
administered after the first examination for the sample as a whole. 
However, correlations for the two separate groups revealed 
significant results only for undergraduates. A groups-by-tests 
repeated measures design was employed to detect between-group 
differences and mathematics skills improvement from pretest to 
posttest. Both groups improved, with greater improvement shown for 
undergraduates, but no conclusions could be drawn regarding the 
reason for the improvement. (Author/JGL) 
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ABSTRACT 

The purpose of this study was to examine and refine a math skills instrument 
selected as a screening tool for prediction of statistics success and as a 
measure of improvement in math skills for students enrolling in introductory 
educational statistics. Alternate forms of a 28- item true-false test were 
administered as a pretest and posttest to 18 undergraduate and 19 graduate 
students enrolled in introductory educational statistics courses during the 
1986 summer term. Cronbach's alpha and corrected item- total correlations were 
computed for the pretest. Item analysis resulted in the elimination of 10 
items from the original instrument. The alpha coefficient for the resulting 
18-item instrument was .75. Total pretest scores were correlated with each of 
three course exams and with course average. Results revealed significant 
correlations with exams administered after the first exam for the sample as a 
whole. However, correlations for the two separate groups revealed significant 
results only for undergraduates. A groups by tests repeated measures design 
was employed to detect between group differences and math skills improvement 
from pretest to posttest. Both groups improved, with greater improvement 
shown for undergraduates. The instrument's utility as a predictive tool and 
as a measure of math skills improvement was discussed. 
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Predicting Success and Measuring Math Skills Improvement 
in an Introductory Statistics Course 
Marcia R. O'Neal, Brad S. Chissom, and Richard F. Ittenbach 

The University of Alabama 

Introduction 

The math skills of undergraduate students enrolling in 
introductory statistics courses vary widely. The completion of 
prerequisite courses is not always a guarantee that students have 
the entering skills necessary for successful completion of course 
requirements in statistics. An instrument which is quickly 
administered and easily graded and which has value in predicting 
success in statistics would be helpful to instructors in their 
efforts to advise students as to whether they should attempt the 
course. Such an instrument would identify those students who do 
not possess the minimum skills necessary to master course 
material, as well as those students who may need extra help 
initially with math operations. 

With the es tablishment of the core curriculum at the 
University of Alabama, an additional use for a math skills test 
has developed. The undergraduate statistics course within the 
area of Behavior ial Studies in the College of Education has been 
granted a math designation. This means that students 
successfully completing the course receive credit toward their 
math requirement needed for graduation . While a course syllabus 
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can be used to document the content of a course, the improvement 
in math skills as a result of a course can be demonstrated only 
through student performance. Thus, courses such is int rodnct ory 
statistics which have been granted math designations need 
instruments which will measure student change in support of the 
math designation • 

The purpose of the present study was to conduct a 
preliminary investigation of a math skills test which would be of 
value in predicting success in an introductory statistics course 
and measuring changes in student math skills from the beginning 
to the end of the course. 

Method 

Subjects 

Participants in the study were students enrolled in the 
undergraduate and graduate levels of the introductory statistics 
course offered by the Area of Behavioral Studies in the College 
of Education at the University of Alabama. A total of 37 
students attending the second Summer term 1986 were included 
initially. Withdrawals and absenteeism resulted in the loss of 
nine subjects , yielding a total of 28 students for the final 
analyses • 

Of the 28 students, equal numbers were enrolled in the 
undergraduate and graduate classes. The undergraduate group 
consisted of 3 males and 11 females, while the graduate class had 
5 males and 9 females. The graduate group was made up almost 



exclusively of Education majors, but undergraduate student majors 
varied, including representation from Colleges of Nursing, Social 
Work, Arts and Sciences, Commerce and Business Administration, 
Education, and Communication* All students in the undergraduate 
course were at least juniors. The majority of students in both 
groups reported limited or no previous computer experience, and 
only three of the 28 students in both groups had a home computer ♦ 
Almost all students reported having had at least an algebra 
course, and over half had taken plane geometry* QFAs of the 
students ranged from 1*50 to 4.00 on a 4-point scale, with the 
graduate group's mean QPA of 3*557 (SD=.38) almost a point higher 
than that of the undergraduate group (Mean=2.68, SD=.60). 
Materials 

The math pretest used in the present study was one which had 
been used in past years for judging the level of entering math 
skills of students in the introductory statistics course. The 
pretest is an adaptation of a test available elsewhere (Walker, 
1934). Although the instrument is more than 50 years old, it was 
chosen because the items appeared to be appropriate for testing 
the math skills of interest; specifically, knowledge of the order 
of operations* These were the skills which, in the opinion of 
the statistics instructor, had given previous statistics students 
particular difficulty in mastering the computational aspects of 
statistics course material. The original test consisted of 24 
i terns , 21 of which were retained . An additional 7 items were 



included, resulting in a 28-item true-false test (Appendix A) on 
which students are required to determine whether values on either 
side of a numerical statement are equalities. 

Statistics exams for each course were developed by course 
instructors. Heavy emphasis was given to com pu tat ions, use of 
statistical tables, selection of appropriate statistical 
analyses, and statistical decision-making. Students were not 
required to memorize formulae, but they were required to carry 
out all computations involved in the use of the formulae. 
Procedure 

Students were given the math pretest at the beginning of the 
five-week statistics course in which they were enrolled. 
Following the pretest, they were given a set of rules for order 
of math operations (Appendix B) . Additional data were collected 
from students on an information sheet (Appendix C) provided at 
the same time as the pretest. An item analysis was performed on 
the pretest responses, yielding item difficulties and item 
discrimination values (based on corrected item-total 
correlations) which were used as a basis for judging which items 
to eliminate. The Kuder-Richardson 20 formula (KR-20; Kuder & 
Richardson, 1937) was used to obtain a measure of the interitem 
consistency among the 18 retained pre test items, and those items 
then became the math pretest items used in later analyses. KR-20 
is the specialized case of coefficient alpha (Cronbach, 1951) 
when items are d ichot omously scored (Nunnally, 1978). 



Math pretest raw scores were computed for each student by 
summing the number of items answered correctly. Students also 
received scores on three major statistics course exams, each of 
which had a maximum point value of 100. Math posttest raw scores 
were obtained by summing the number of items answered correctly 
on an alternate form of the revised pretest. The unrevised 
posttest (containing all 28 items) can be found in Appendix D. 

To determine the magnitude and direction of the relationship 
between math skills and statistics course performance, Pearson 
product-moment correlation coefficients were computed for the 
math pretest with exam 1, exam 2, exam 3, and exam average for 
each of the two groups separately and for the 28 students 
combined • 

Improvement in math skills was assessed through a groups by 
tests repeated measures analysis of, variance design. The use of 
this design allowed for the exa:vination of differences between 
groups as well as change from pretest to posttest. This was of 
particular interest because it was expected that the graduate 
students, by virtue of their more extensive college experience, 
would achieve higher pretest scores, and this might affect 
results • 

Results 

Item difficulties and item discrimination values were 
computed for the 28 -it em math skills pretest results of the 37 
students who were administered the test. Several values were 
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used as general guidelines in considering an item for retention 
or elimination ♦ Chissom, McLean, and Hoenes (1980) have 
suggested that good items are those with discrimination indices 
of .40 or greater and difficulty indices between .40 and *70* 
According to Nunnally (1978), items whose discrimination values 
equal or exceed .30 are generally considered good. 

Item analysis results led to the removal of eight items with 
the lowest ( ♦ 10 and below) item- total correlations first ♦ 
Subsequent analysis resulted in the removal of two more items 
until 18 items remained* Some of these items had difficulty or 
discrimination indices falling outside the values suggested 
above* Such items were examined and then retained to maintain an 
instrument of adequate length and for which the internal 
consistency value would not be reduced from that of the original 
28-item instrument* Furthermore, as Mehrens and Lehmann (1984) 
have indicated, there are e number of reasons for retaining items 
even when discrimination indices are low* A low discrimination 
index may result because an item is very easy or very difficult, 
and easy items may be important to retain to aid student 
motivation* As long as the item discriminates positively, it is 
contributing to the measurement of student competency* 
Coefficient alpha for the 18-item instrument was *75, as compared 
with *73 for the original 28-item instrument* Nunnally (1978) 
indicated that reliability coefficients of *70 or higher are 



sufficient for early stages research. Item analysis results 
can be found in Tables 1 and 2. 
Table 1 

Item Analysis Results for 28 Math Skills Items for 37 Students 



Item 

nm H p t* 


1 1 em 
Dif f icul tv 


1 1 em 

Discrimination 

JL «^ ^» JL UA JL L L O l» JL V/ L L 


1* 


1.00 


**** 


2* 


0.22 


0.10 


3 


0.65 


0.62 


4* 


1 . 00 


**** 


5* 


1 .00 


**** 


6 


0.81 


0.24 


7 


0.51 


0.51 


8 


0.59 


0.25 


9 


0.32 


0.25 


10 


0.92 


C . 14 


11 


0.86 


0.19 


12* 


0.81 


0.07 


13 


0.84 


C.42 


14 


0.86 


0.15 


15 


0.70 


0.38 


16 


0.51 


0.24 


17* 


0.54 


0.04 


18* 


0.78 


0.05 


19 


0.84 


0.32 


20 


0.81 


0.15 


21 


0.46 


0.34 


22* 


0.78 


0.04 


23 


0,86 


0.39 


24 


0.78 


0.51 


25 


0.78 


0.40 


26*a 


0.92 


0.19 


27 


0.81 


0.24 


28*a 


0.65 


0.46 



* indicates items eliminated* 

**** indicates items for which correlations could not be 
computed • 

a* These items were ultimately eliminated because a subsequent 
item analysis resulted in very low or negative item-total 
correlations • 
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Table 2 

- Item Analysis for 18 Math Pretest Items for 37 students 



Item Item Item 
Number Difficulty Discr iminat ion 



1 0.65 0.52 

2 0.81 0.26 

3 0.51 0.52 

4 0.59 0.16 

5 0.32 0.19 

6 0.92 0.22 

7 " 0.86 0.18 

8 0.84 0.44 

9 0.86 0.23 

10 0.70 0.37 

11 0.51 0.36 

12 0.84 0.36 

13 0.81 0.17 

14 0.46 0.38 

15 0.86 0.44 

16 0.78 0.52 

17 0.78 0.38 

18 0.81 0.24 



Correlations and analysis of variance results were based on 
a total of 28 students; 14 in each of Che two statistics classes. 
Of the 9 students excluded, 5 withdrew from the courses, and 4 
were absent from one of the test administrations. Those students 
who withdrew or were absent did not differ on demographic 
variables from the 28 students included in the analyses. T-tests 
revealed that students who withdrew from the course did not 
differ on pretest results from students who participated. 

Math skills pretest scores ranged from 5 to 18 for the 28 
students. Their math skills posttest scores ranged from 12 to 
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18« On exam 1, scores ranged from 67 to 100* For exam 2, the 
range was 35 to 100, and for exam 3, scores varied from 23 to 
100* The average of the three tests ranged from 43.33 to 99, 
Means and standard deviations on the math skills tests and 
statistics measures for the 28 students and for the two separate 
groups are shown in Table 3« 
Table 3 

Means and Standard Deviations for Math Skills and Statistics 
Exams 



Graduate Course 



Test 


Mean 


SD 


Mean 


3D 


Mean 


SD 


Math 














Pretest 


11.36 


3.65 


14.07 


2.95 


12.71 


3.54 


Math 














Posttest 


15 .29 


1.68 


16.07 


2.02 


15 .68 


1.87 


Exam 1 


90.71 


8.11 


88.71 


9.19 


C9.71 


8.57 


Exam 2 


75.21 


18 .23 


85.43 


15 .97 


80 .32 


17 .60 


Exam 3 


75.29 


21.42 


79.57 


20.40 


77 .43 


20.64 


Exam 














Average 


80.41 


11.96 


84.57 


13 .72 


82 .49 


12.31 


The 
obtained 


18-item math pretest yielded 
by summing correct responses 


a score 
to the 


for each student, 
18 items. Math 



pretest scores were then correlated with each of the statistics 
exams and with the exam average for students in each of the two 
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courses and for all 28 students combined. These correlations are 
shown in Table 4. 
Table 4 

Correlations of Math Pretest Scores with Statistics Measures 



Test 




Undergraduate 
Course (N=14) 


Graduate 
Course (N=14) 


Combined 
Course (N=28) 


Exam 


1 


-.26 


.33 


-.03 


Exam 


2 


.62** 


.27 


.53** 


Exam 


3 


.65** 


. 12 


.43* 


Exam 


Average 


.65** 


.24 


.47** 



* £ < .05 ** £ < .01 

A significant relationship did not exist between math pretest 
scores and exam 1 scores for either group. A statistically 
significant relationship did exist for math pretest scores and 
exam 2, exam 3, and exam average for the undergraduate class but 
not for the graduate class. Using math pretest score as the 
single predictor variable with course average as the criterion, 
the slope and intercept were computed for undergraduate course 
students. The prediction equation shown below was the result. 

AAA 

y - B 0 + B f x = 56 .26 + 2. 13x 
Using this prediction equation, the predicted exam average for a 
s tudent who obtained a score of 6, for example , on the math 
pretest would be 69.04. 
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Prior to performing the analysis of variance , the 
assumptions associated with the design were tested* Testing for 
homogeneity of variance, it was found that the assumption was not 
violated. The obtained Fmax value of 4.7 did not exceed the 
critical value of 6.5 (Fmax,c<= # 01 , k=4 , df=13). Tests of 
additivity revealed violations of this assumption for both 
groups. For the undergraduate group, an F value of 12.95 was 
obtained; for the graduate group, the obtained F value was 6.62. 
Both of these value exceeded the critical value of 4.75 at the 
.05 level. Despite the violations of the assumption of 
additivity, the analysis continued as planned since this type of 
analysis is robust in the face of such violations. The nature of 
the violation would simply make the statistical test more 
conservative. 

Analysis of variance results are found in Table 5. 
Table 5 

ANOVA Summary Table for Course-by-Tes t Repeated Measures 



Source SS df MS F 



Course 42.88 
Error Between 299.46 



1 

26 



42.88 
11.52 



3.72 



Test 

Test x Course 
Error Within 



123.02 
13.02 
76.46 



1 
1 

26 



123.02 
13.02 
2.94 



41.83 
4.43* 



*F(1,26) > 4.23; £ < .05 



Results revealed a statistically significant interaction between 
course ant 1 test. Figure 1 depicts the interaction graphically. 




Figure 1. Interaction between groups and measures. 



x - undergraduates 
o - graduates 
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Since a statistically significant interaction was found, simple 
main effects were examined. The results of this analysis are 
shown in Table 6. 
Table 6 

Simple Effects for Course and for Test 



Source 


SS 


df 


MS 


Error Term 


F 


Course 












Course @ 
Pretest 


51.57 


1 


51 .57 


6.96 


7.41* 


Course @ 
Posttest 


4.32 


1 


4.32 


6.96 


.62 


Test 












Test @ 
Undergrad 


108 .04 


1 


108 .04 


2.94 


36 .74** 


Test @ 
Graduate 


28 .00 


1 


28 .00 


2.94 


9.52** 



*F(1,54) > 4.03; £ < .05 **F(1,27) > 4.21; £ < .05 

Results indicated a statistically significant difference between 
the two groups at pretest but not at posttest. There was also a 
statistically significant difference between pretest and posttest 
for both groups. The Eta squared value calculated for the 
analysis was found to be .32, indicating that 32% of the 
variability was due to the interaction between groups and tests . 

Discussion 

Results of the present study indicate that for this group of 
students, there was a statistically significant relationship 
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between math skills scores and statistics course performance for 
students in the undergraduate course but not for graduate 
students. Although the study involved a small sample size, the 
results provide preliminary evidence to indicate that the revised 
math skills test has the potential for use in predicting 
performance in the undergraduate statistics course. 

Analysis of variance results do reflect improvement from 
pretest to posttest for students in both courses, thus providing 
evidence for the sensitivity of the instrument to change. 
However, because of the nature of the research design, no 
conclusions can be drawn regarding the reason for improvement. 
Further, it is not known whether the change demonstrated in the 
five-week course will be evident in courses conducted during the 
regular 16-week school semester. 

Research is currently under way using a larger sample for 
comparison of results with present findings. Item analysis 
results will be used for revision of the 28-item instrument, and 
correlations will again be used in an effort to accumulate 
further evidence of the predictive capabilities of the revised 
math skills test. 

Measurement of math skills improvement is planned which will 
involve an experimental-control group design, thus addressing the 
threats to internal validity, such as history (Campbell & 
Stanley, 1963), not addressed in the present study. 
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BE? 445 
PRETEST* 
EDUCATIONAL STATISTICS 



INSTRUCTIONS : 



8. 25 t 2 + 3 = 5 



10. 


(4 


+ 


8) 






3 




11. 


(4 


+ 


8) 




(3 


X 


2) 


12. 


(4 


X 


8) 




(3 


X 


2) 


13. 


(4 


+ 


3) 




(3 


+ 


2) 


14. 


(6 


X 


5 3 



3 x 2 



3 

2x5x7 



Indicate in the blank next to the item whether each statement is 
true or false. 

1. 3 + 4 + 7 = 7 + 3 + 4 15. (4 x 5) 4 5 

3 " 3 x 3 

2. 3 + 42 7 5 - 9 

3. 12(3/4) = 3(12/4) 

4. 12 v 3 + 2 = 22/5 

5. 64 v (IS x 2) = 8 " 

19 . 4 + 3 _ 4 . 

6. (3/4) 2 « 9/16 3 ~ 3 * A 



16. V 25 + 16 =5+4 

17. 5(2 - 1/15) = 10 - 1/3 

18. 15 * 3 + 2 = 7 



7. (2/3) (5/4) (3/10) = 20. 36+5 = g + g 

(2/4) (5/10) (3/3) 



6 



21. 1/2 (5x7)= (5/2) (7/2) 

22. 6 t 2+1=2 

1/4-3/8 23. (7-2)2=25 



9. 1/4 (1 - 3/2) = 

2 - 



4 8 24. 3(4 x 5) = 12 x 15 

= 3~ + 1 

25. 15 + 9/3 + 3 = 21 

4 8 

= 3 + 2 ■ 26. (17 - 5)/(8 + 4) = 1 



_27. (2/7)2 = 4/49 
28. 7(0 + 2)/(3 + 2) = 14 



* From Walker, Helen, Mathematics Essential for Elementary 
Statistics. 
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BER-5A0 

Statistical Procedures in Education 
EXPLANATION AND RULES* 



1 # The order in which nunb'>rs are added does not affect the result. 

2. The order in which numbers ar* multiplied does not affect the result. 

3. If both multiplication an, 5 addition (or subtraction) are indicated, 
the multiplication should be performed first unless parentheses, or 
other symbols of grouping, indicate otherwise. 

A. If both division and addition (or subtraction) are indicated, the 
division should be performed first unless parentheses, or other 
symbols of grouping, indicate otherwise. 

5. If both multiplication and division are indicated, or more than one 
division, the expression may be ambiguous unless its meaning is 
clarified by parentheses or other signs of grouping. 

6. Parentheses, braces, and brackets ( ), { \ , may be used to 
show that whatever is enclosed within them is to be treated as a 
single number. 

7. If the bar of a fraction has the same effect as parentheses, the 
numerator being treated as a single number and the denominator as 
a single number. 

8. A radical sign has the effect of parentheses, the expression under 
the radical sign being treated as a single number. 

9. In an equation the « is not destroyed if 

a. same number is added to both members 

b. same number is subtracted to both members 

c. same number is multiplied by both members 

d. same number is divided into both members 



* From Walker, Helen, Mathematics Essential for Elementary Statistics. 
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Information Sheet 

Name : 
Address : 



Phone : 

S tudent Number : 
Major : 

Class Standing: 
QPA: 

Check any of the following math courses you have had either in 
high school or in college: 

Basic Math 

Algebra 

Plane Geometry 

Solid Geometry 

Trigonometry 

Calculus 

In what year was your last math class? ______ 



What math course (topic) was it? 



How much experience have you had using computers? (circle one) 
None Limited Moderate Extensive 

Do you own a home computer? (circle one) 
Yes No 
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BER 445/540 
POSTTEST 
Educational statistics 

NAME : 



INSTRUCTIONS : Indicate in the blank next to the item whether each 
statement is true or false. 



1. 9 + 6- 2- 6- 2 + 9 

2. 48 z 6 + 2 - 6 

3. 75 



4. 72 7 2 + 4- 12 



_5. 5 + 63 7 3 



68 



6. (1/3)' 



1/9 



15. 5x6 5_ 6_ 
4 " 4 X A 



I6."y36 -25-6-5 



17 . 



18. 25-3+8 

19. 12+2 



30 



20. 48+5 



= 8+5 



- 7 - m&mm 



21. 1/3 (6 x 4) - 2(4/3) 



8. 


*8 f 


(2 x 


3) - 8 


22. 


4-2 + 2-4 


9. 


£(32 


+ 3) 


-4 + 3 


23. 


(11 - 3) 2 - 64 


10. 


9 + 

3 


2 


3 + 2 


24. 


2(5 x 3) - 10 x 6 


11. 


5 + 


7 




25. 


12 + 6/2 +4-19 




(2 x 


3) ~ 


| +| . 






12. 


(5 x 


7) - 


! «*' 


26. 


(17 + 5)/(8 + 3) - 




(2 x 


3) 








13. 


(5 + 




i + 1 


27 . 


(2/8) 2 - 4/64 




(2 + 


3) 


2 3 






14. 


(15x4x7) 
5 


-3x4x7 


28. 


5(6 + 2)/(7 - 2) - 



* From Walker, Helen, Mathematics Essential for Elementary 
Statistics. 



